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In  areas  of  risk  assessment  ranging  from  terrorism  to  health,  safety,  and  the  environment,  authoritative  guidance 
urges  risk  analysts  to  quantify  and  display  their  uncertainties  about  inputs  that  significantly  affect  the  results  of 
an  analysis,  including  their  uncertainties  about  subjective  probabilities  of  events.  Such  "uncertainty  characteri¬ 
zation"  is  said  to  be  an  important  part  of  fully  and  honestly  informing  decision  makers  about  the  estimates  and 
uncertainties  in  analyses  that  support  policy  recommendations,  enabling  them  to  make  better  decisions.  But  is 
it?  Characterization  of  uncertainties  about  probabilities  often  carries  zero  value  of  information  and  accomplishes 
nothing  to  improve  risk-management  decisions.  Uncertainties  about  consequence  probabilities  are  not  worth 
characterizing  when  final  actions  must  be  taken  based  on  information  available  now. 

"But  there  seemed  to  be  no  chance  of  this,  so  she  began  looking  at  everything  about  her  to  pass  away 
the  time." 

Lewis  Carroll,  Alice  in  Wonderland 
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Quantitative  risk  analysts  are  blessed  today  with 
a  wealth  of  advice  and  instruction  designed  to 
help  them  live  more  useful  lives  and  to  communicate 
their  speculations  and  uncertainties  to  policy  makers 
with  computer-aided  clarity  Where  previous  gener¬ 
ations  were  restrained  by  the  conviction  that  some 
uncertainties  did  not  matter  (for  example,  because 
zero  times  X  is  zero,  no  matter  how  great  the  uncer¬ 
tainty  about  X),  today's  analysts  have  been  liberated 
and  empowered  by  uncertainty-analysis  software. 

"Your  estimates  cannot  be  more  precise  than  their 
most  uncertain  component,"  they  are  told  (Office 
of  Management  and  Budget  (OMB)  2003).  The  duty 
of  every  good  analyst  is  to  characterize  uncer¬ 
tainties  and  to  present  them  to  decision  makers. 
Plentiful  guidance  is  available  from  and  for  regula¬ 
tory  analysts;  "Apply  a  formal  probabilistic  analysis 


of  the  relevant  uncertainties — possibly  using  simula¬ 
tion  models  and  / or  expert  judgment  as  revealed,  for 
example,  through  Delphi  methods.  Such  a  formal  ana¬ 
lytical  approach ...  is  required  for  rules  that  exceed 
the  $1  billion  annual  threshold.  ...You  should  make 
a  special  effort  to  portray  the  probabilistic  results — 
in  graphs  and/or  tables — clearly  and  meaningfully" 
(ibid).  Inspiring  words  to  aspire  to . .  .but  can  we  live 
by  them? 

Perhaps  the  desire  to  tell  all  can  be  overdone.  Are 
uncertainties  about  risks  always  worth  characteriz¬ 
ing?  Are  uncertainties  about  these  uncertainties  useful ? 

Quantifying  a  Probability 

Suppose  that  your  client,  a  government-agency  func¬ 
tionary  who  is  responsible  for  important  risk  assess- 
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merits,  asks  you  (the  expert)  for  your  subjective 
degree  of  belief  (let's  call  it  "p")  that  a  particular  tech¬ 
nology  will  become  available  to  the  general  public 
within  three  years.  If  it  does,  then  a  distribution  over 
consequences  (sometimes  called  a  "risk  curve")  to 
our  country  can  be  calculated  using  a  standard  and 
well-validated  model;  if  it  does  not,  then  this  model 
can  be  used  to  produce  a  different  distribution  over 
consequences. 

After  doing  whatever  it  is  that  made  you  an 
expert,  you  answer,  "p  =  0.35."  This  is  too  simple 
for  your  client,  who  reproachfully  reminds  you  that 
point  estimates  are  for  chumps,  and  that  "Monte 
Carlo  and  other  probabilistic  methods — simulating 
a  distribution  of  the  results  by  randomly  draw¬ 
ing  from  the  probability  distributions  of  input  vari¬ 
ables  and  repeating  the  analysis  numerous  times" 
are  now  de  rigueur  (US  Environmental  Protection 
Agency  (EPA)  1999).  The  client  shows  you  copies 
of  uncertainty-analysis  policies  from  other  agencies 
and  agency  heads,  past  and  present,  emphasizing 
that  "First,  we  must  insist  on  risk  calculations  being 
expressed  as  distributions  of  estimates  and  not  as 
magic  numbers"  (National  Research  Council  (NRC) 
1994,  p.  160).  The  client  further  admonishes  you  that 
it  is  key  to  distinguish  between  epistemic  and  aleatory 
sources  of  uncertainty  in  your  estimate. 

You  look  at  your  "p  —  0.35"  and  feel  embarrassed. 
You  are  not  sure  what  epistemic  uncertainty  is,  and 
you  do  not  know  whether  aleatory  uncertainty  will 
turn  out  to  matter,  but  you  are  pretty  certain  that  your 
response  of  0.35  is  going  to  need  some  enhancements. 

"Can  you  take  a  few  months,"  asks  your  client,  "to 
recognize  that,  because  you  are  uncertain  about  its 
value,  the  probability  you  just  gave  me  is  in  fact  a  ran¬ 
dom  variable  P,  and  therefore  please  produce  a  distri¬ 
bution  for  it?  Then  I  can  present  this  to  my  agency's 
decision  makers  with  appropriately  detailed  quantita¬ 
tive  support." 

You  agree  to  do  so  (perhaps  feeling  that  life  offers 
too  few  opportunities  to  get  paid  for  such  deep 
introspection). 

Elicitation  of  Uncertainty  About 
a  Probability 

Specialists  in  expert  elicitation  train  you  on  calibration 
and  bias  and  the  best  approaches  for  eliciting  subjec¬ 


tive  distributions.  You  oblige  them  and  produce  /(p), 
a  distribution  for  P.  This  distribution,  not  surprisingly, 
turns  out  to  have  an  expected  value  of  E(P)  —  0.35! 
More  importantly,  during  this  time,  you  realize  that  a 
critical  factor  you  subjectively  used  in  coming  up  with 
the  distribution  over  P  (and  the  value  0.35  in  the  first 
place)  was  whether  a  particular  chemical  process  can 
be  stabilized  within  the  next  six  months. 

Pleased  with  your  efforts,  your  client  takes  your 
resulting  subjective  uncertainty  distribution  f{p),  hav¬ 
ing  expectation  of  0.35,  and  generates  plots  of  it 
using  various  graphics  programs  and  a  new  clus¬ 
ter  of  supercomputers  recently  procured  to  facilitate 
fuller  and  more  responsible  uncertainty  characteriza¬ 
tion  in  these  troubled  times.  This  software  displays 
quintiles  of  your  uncertainty  distribution,  prepares 
bar  charts  with  multicolored,  nested,  subjective  confi¬ 
dence  intervals,  and  fits  parametric  distributions  and 
cubic  splines  to  your  results.  In  addition,  it  almost 
automatically  adds  graphical  appendices  to  your  orig¬ 
inal  report  (that  was  a  mere  e-mail  with  the  unassum¬ 
ing  estimate  "p  —  0.35"  in  its  text)  to  show  how  rich 
and  deep  your  reflections  about  that  0.35  really  are. 

With  the  remaining  budget — your  client's  budget 
this  year  for  expert  elicitation  and  uncertainty  anal¬ 
ysis  seems  almost  limitless  as  agencies  responsible 
for  addressing  issues  ranging  from  climate  change 
to  the  war  on  terror  realize  how  impressive  uncer¬ 
tainty  analyses  look — he  enhances  the  graphs  to  show 
deciles  and  other  quantiles  of  your  distribution  f(p), 
initiates  a  research  project  at  a  major  university  to 
develop  resampling  methods  to  better  estimate  the 
parameters  of  the  distributions  that  approximate  /(p), 
and  convenes  a  four-day  workshop  to  discuss  the 
importance  of  clearly  displaying  uncertainties  about 
uncertainties  about  risks. 

Zero  Information  Value  of  Uncertainty 
Characterization 

At  the  workshop,  you  fall  into  a  somnolent  reverie. 
You  realize  that  all  of  this  well-meant  activity 
has  added  precisely  nothing  (value  of  information 
(VOI)  =  0)  to  your  initial  assessment  of  p  —  0.35.  It 
seems  to  be  a  pure  waste  of  resources  from  the  stand¬ 
point  of  improving  or  informing  decisions. 

Why?  Because,  at  the  end  of  this  tortuous  process, 
you  are  still  saying  neither  more  nor  less  than 
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that  you  judge  the  probability  that  a  particular 
technology  will  become  available  to  the  general  pub¬ 
lic  within  three  years  to  be  0.35.  No  amount  of 
additional  "elicitation"  and  no  artistic  rendering  of 
your  uncertainty  about  that  number  changed  it  in 
the  slightest  or  added  any  value  to  it.  The  distribu¬ 
tion  of  your  uncertainty  about  that  probability  might 
be  interesting  from  a  philosophical  point  of  view, 
and  might  even  be  useful — but  only  if  you  were  to 
be  given  additional  information  about  the  progress 
in  stabilizing  the  critical  chemical  process.  That  is, 
the  VOI  about  the  chemical  process  stabilization 
might  be  greater  than  zero.  However,  you  were  not 
given  this  information;  indeed,  if  you  had  been  given 
it,  your  whole  assessment  would  have  changed  in 
response. 

The  bottom  line  is  that  you  have  been  asked  for 
a  probability,  given  the  information  and  insights  you 
have  now  (not  what  you  might  have  learned  had 
additional  information  been  available).  And,  as  you 
were  told  in  your  introductory  probability  course,  this 
probability — the  expected  value  of  a  binary  indicator 
for  the  event — is  simply  a  number,  not  a  distribution. 

The  Brave  New  World  of  Uncertainty 
Characterization 

You  awaken  from  these  reflections  to  hear  the  work¬ 
shop's  plenary  speaker — famous  for  his  work  at  the 
boundary  of  neuroeconomics  and  risk  perception — 
suggesting  that  "yes,"  "no,"  and  "maybe"  are  bio¬ 
logically  meaningless  concepts,  unworthy  of  use  in 
scientific  uncertainty  analysis,  and  that  continuous 
distributions  of  values  (corresponding  to  dopamine 
gradients)  are  essential  for  biologically  realistic  risk 
communication  between  parts  of  the  brain.  You  know 
this  will  appeal  to  the  many  subject  matter  experts 
you  have  met  who  refuse  to  commit  to  such  crisp 
terms  as  "always"  or  "never." 

You  sit  up  when  the  speaker  shows  an  exciting 
film  clip  of  functional  magnetic  resonance  images 
(fMRI)  of  the  posterior  cingulate  cortex  of  your  client 
agency's  risk  manager  while  he  is  being  offered  rein¬ 
forcements  of  (1)  fruit  juice  or  (2)  video  clips  of  ani¬ 
mated  3D  uncertainty  distributions  for  risk  estimates. 

You  are  a  bit  disappointed  to  hear  that  it  was 
ambiguous  which  reinforcement,  if  either,  more 
effectively  primed  the  risk-management  brain  cen¬ 


ters  (but  the  false-color  images  were  terrific!),  and 
you  are  chagrined  to  reflect  that  all  you  really  had 
to  say  about  that  technology  becoming  available 
was  that  you  thought  it  had  a  35  percent  chance. 
(Admittedly,  the  visualization  now  scrolling  across 
the  huge  auditorium  screen,  showing  3D  anima¬ 
tions  of  your  uncertainty  distribution  as  approxi¬ 
mated  from  different  bootstrap  samples,  and  using 
alternative  statistical  models  and  a  Bayesian  model¬ 
averaging  postprocessing  step,  almost  convinces  even 
you  that  there  must  be  more  to  the  story  than  that.) 

Epilogue 

You  have  met  some  nice  people  who  clearly  know 
a  lot  about  expert  probability  elicitation  and  firmly 
believe  that  it  will  empower  better  decisions.  They 
did  seem  a  little  vague  about  how,  but  reassured  you 
that  current  scholarship  holds  that  "the  onus  is  on 
the  communicators  of  the  probabilistic  information  to 
help  people  find  better  ways  of  using  the  information, 
in  such  a  manner  that  respects  the  users'  autonomy, 
full  set  of  concerns  and  goals,  and  cognitive  perspec¬ 
tive"  (Patt  and  Dessai  2005,  p.  437). 

You  eagerly  anticipate  working  next  year  with 
the  risk-communication  team  and  its  fMRI  machine 
to  make  sure  that  your  probability  assessments  are 
communicated  with  proper  respect  for  all  possible 
decision-maker  concerns,  goals,  and  perspectives. 

And  you  can  hardly  wait  to  meet  the  decision 
makers  who  will  be  funding  this  uncertainty-about- 
uncertainty  characterization  effort  for  many  years  to 
come.  Will  they  like  your  work?  Will  they  remem¬ 
ber  to  analyze  the  robustness  of  their  decision  with 
respect  to  p?  The  client  has  been  effusive  in  his  com¬ 
pliments  of  the  thick  and  colorful  report  you  helped 
to  produce,  and  has  told  you  that  the  decision  mak¬ 
ers  were  delighted  with  the  graphics  (especially  the 
smooth  gradations  of  colors  to  suggest  mixtures  of 
aleatory  and  epistemic  uncertainties).  Will  they  con¬ 
tinue  to  fund  it?  You  are  not  sure.  But  you  are  com¬ 
fortable  giving  it  a  probability  of  0.99. 
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